Use asynchronous aggregation queries#2030
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the BigQuery aggregation process to run asynchronously, introducing new methods for job submission and status polling across the BigQueryExecutor, AggregationUtils, and the ingestion workflow. Feedback identifies a logic error in get_jobs_status where a 'FAILED' status could be incorrectly overwritten by 'RUNNING' due to iteration order. Additionally, it is recommended to restore the 1800-second timeout in the workflow YAML to prevent premature timeouts during the initial job submission phase if the import list is large.
vish-cs
force-pushed
the
aggregation
branch
4 times, most recently
from
453ec4c to
22fb1f4
Compare
|
Putting this here but we already discussed in chat - pls dont submit this. It would break the DCP setup. In general we should always strive to keep backwards compatibility, move the traffic over, then tear down the old code. We've been getting away with not doing that because it was just our instance, but this is definitely changing now :) |
3 participants
No description provided.